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The Circuit Maintenance System (cms) -IB has been developed to 
provide operational, administrative, and data base support to the 
trunk maintenance areas of the No. 4 ess. In this paper, we investigate 
the reliability of the cms -IB under the most stringent situation where 
six No. 4 esss are supported simultaneously. Our study results indi- 
cate that, with its current hardware arrangements, software structure, 
and administrative procedure, the cms -IB can meet the reliability 
objective. Furthermore, the reliability of the cms-IB is insensitive to 
an incremental change of its operating characteristics pertaining to 
the above three categories. We also provide a field-of-use guideline 
whereby the reliability of the cms -IB can best be upgraded, if neces- 
sary. This guideline can be used by each cmsI-B site to decide 
whether its maintenance contract meets the response time require- 
ment or its disk drive system needs to be replaced by a more reliable 
counterpart. 

I. INTRODUCTION 

The Circuit Maintenance System (cms)-IB has a multiple office 
feature and can support up to six No. 4 esss* (for a description of No. 
4 ess and cms, see Refs. 1 and 2). For a high degree of reliability, the 
cms- IB employs a duplex computer system with an operating system 
designed to have fast reboot time and backup recovery capabilities. 

In addition to its hardware and software structures, the reliability of 
the cms- IB depends on the proficiency of the on-site corrective main- 
tenance activities. Switching over from a faulty to a standby system, 
rebooting the system in the event of a software error, and repairing 



* Since the total number of No. 4 ess trunk terminations supported by a cms-IB 
cannot exceed 160K, not more than one of these No. 4 esss is full-sized. 
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the faulty system are some maintenance activities required to bring 
the cms- IB back to normal operation. 

In this paper, we quantify some of the significant factors governing 
the cms- IB reliability. More importantly, we evaluate whether the 
level of reliability attained by the cms- IB would have any adverse 
impact on the No. 4 ess served by it. We determine whether the 
current hardware arrangement, software structure, or administrative 
procedures are adequate from the No. 4 ess's point of view. Should 
they be inadequate, we recommend some feasible means of upgrading 
the reliability of the cms- IB. 

The remaining portions of this paper are organized as follows. 
Section II provides a simplified description of the hardware arrange- 
ment of the cms- IB. Because the cms- IB can be viewed as consisting 
of a duplex and a simplex subsystem in tandem, the reliability models 
for these two subsystems are addressed in Sections III and IV, respec- 
tively. A reliability objective of the trunk maintenance operation of 
the No. 4 ess is established in Section V. Section VI summarizes the 
input data required for the numerical computation of the reliability 
model. In Section VII, the reliability of the cms- IB is measured against 
its objective as determined by the needs of No. 4 ess. Sensitivity 
analyses are also presented. A field-of-use guideline by which the 
reliability of the cms- IB can be upgraded is discussed in Section VII 
and, finally, conclusions are drawn in Section VIII. 

II. CIRCUIT MAINTENANCE SYSTEM-1B 

Figure 1 is a simplified cms- IB hardware block diagram. The cms- 
IB consists of a duplicated Digital Equipment Corporation (DEC) pdp 
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Fig. 1— Simplified cms-IB block diagram. 
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11/70 processors, core memory, and mass storage devices in the form 
of disk drives (dd) and magnetic tape drives (mtd). The cms-IB 
interconnects to and communicates with a No. 4 ess through the data 
circuits and the line multiplex unit (lmu). The processor switch 
performs the function of switching the data circuits to the active lmu. 
For the sake of simplicity, other peripheral devices, such as the 
processor-controlled alarm circuits and the real-time clock, are not 
shown. 

Primary system storage of the cms- IB programs and data needed to 
facilitate trunk maintenance operations is provided by the disk con- 
trollers and disk drives. Since two disk drives are required for each 
No. 4 ess, as many as 12 disk drives are being actively used. As the 
disk drive system is not duplicated, two hot spare or powered-up 
standby drives are provided to maintain a high level of reliability. 

The cms- IB can be physically divided into two independent subsys- 
tems, the duplex and the simplex systems. The simplex system repre- 
sents the disk drive system while the duplex one signifies the rest of 
the cms- IB. These two independent subsystems are connected in 
series, and a malfunction in either subsystem will affect the normal 
operation of the cms-IB. Mathematically, if the availabilities of the 
duplex and the simplex systems are denoted by Ad and A a , respectively, 
the availability of the cms- IB is given by 

A = A d A s . (1) 

Conversely, the unavailability of the cms- IB is given by 

17=1 -A, (2) 

which signifies the fraction of downtime expressed in hours per year or 
in minutes per day. 

The system unavailability of (2) will henceforth be used as a measure 
of the degree of reliability of the cms- IB. To quantify it, the individual 
availabilities of the duplex and the simplex systems must first be 
determined. 

III. DUPLEX SYSTEM 

The normal operation of this subsystem can be interrupted by either 
a hardware or a software problem. The hardware-related problems 
include those associated with the central processor units, the memory 
banks, the magnetic tape drives, the disk controllers, or the line 
multiplex units. In other words, if the failure rate of the ith hardware 
unit is At j the overall hardware failure rate Xh of the duplex system is 
computed as 

\h = 2 K (3) 

where /signifies the total number of individual hardware units. 
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When a hardware failure occurs, the cms- IB can be brought back to 
its normal operation almost instantly by switching from the faulty to 
the standby unit, if one is available. The average lengths of time taken 
by a repair person to report to the cms- IB site and to repair the faulty 
unit are called the mean response time, l//i>, and the mean repair time, 
1/fir, respectively. 

Unlike the hardware problems, software problems are less well- 
defined and their failure and repair statistics are not readily available. 
In general, software problems of a mature system are expected to be 
diagnosed and rebooted within a short time interval. Rectification of 
some software problems such as a debilitating data base requires more 
than a reboot action and therefore a longer time period. However, 
characterization of these less frequent but more severe software prob- 
lems is presently not well understood and is therefore excluded from 
our reliability study. With this assumption, the software failure rate is 
represented by A s and the corresponding mean reboot time by 1//1&. 

3. 1 System states 

In view of the front-end and the backup system arrangement and 
the possible occurrence of a hardware and a software failure, the 
duplex system has a large number of possible system states. The 
various states of the duplex system are summarized in Table I. Both 
the front-end andjt>ackup systems as well as the software are operative 
in State 1. While the software is still in operation, the backup and the 
front-end system become inoperative in States 2 and 3, respectively. 
However, the cms- IB is still in its normal operation in these two states. 
The remaining states signifity other possible combinations. 

Based on the states defined in Table I, the cms- IB is available in 
States 1 to 3 and unavailable in States 4 to 8. 

3.2 State transitions 

All the possible transitions interconnecting the eight states are 
diagrammed in Fig. 2. Three possible transitions emanate from State 
1 as a result of a front-end, a backup, or a software failure. States 2, 3, 
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Fig. 2— State transition diagram of the duplex system. 

and 6 can return to State 1 through repair or reboot action. States 4, 
5, 6, and 8 indicate a software failure and can return to their respective 
preceding states by rebooting. Both the front-end and the backup 
systems suffer a hardware failure at States 7 and 8, wherein the cms- 
IB would be inoperative for a long period of time. 

3.3 Mathematical representation 

For mathematical simplicity, we assume that both the hardware and 
the software failures are independent and Poisson distributed. We also 
assume that the times to switchover, to respond to a service call, to 
repair, and to reboot are exponentially distributed. Using these as- 
sumptions and denoting p, as the steady-state probability of being in 
State i of the duplex system, we can derive from Fig. 2 a set of first- 
order differential equations: 

p\ = -(2X h + \s)pi + /x r p 2 + HrPa + HbPs 

P2 = XhPl — (A/, + A s + Hr)p2 + HbP6 + llrPl 
P3 = XhPi - (Xh + As + jU r )/? 3 + jU&p 4 
p 4 = A,/?3 - flbPi + \hP5 

p\ = XsP\ - (2X h + [ib)ps 

p e = X s p 2 + XhPn - (Xh + Hb)pe 

Pi = XhP2 + XhPa — (X s + Hr)pi + [IbPs 

Ps = A/,p 6 + X s p: - HbPs, (4) 

where 

i p. = i. 

(=1 

Solving these eight differential equations using standard techniques, 
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we compute the availability of the duplex system by summing the "up" 
probabilities as 

A d =% Pi . (5) 

i=l 

IV. SIMPLEX SYSTEM 

The disk drive system is said to be inoperative whenever a disk drive 
fails or some data are lost due to such mishaps as a headcrash. The 
failure and headcrash rates of the disk drives, active and spare, are 
denoted by A^ and X c , respectively. It should be pointed out that, unlike 
the duplex system, when the simplex system is inoperative, only one 
No. 4 ess is affected. 

In the event of a disk drive failure, the simplex system can be 
brought back to normal operation by moving the disk pack involved 
from the faulty disk drive to a spare one, if one is standing by. The 
average amount of time required to perform this changeover is called 
the mean switchover time, l//x s . 

In the case of a headcrash, the lost data must first be reconstructed 
before switching over to a spare disk drive. The average length of time 
required to rebuild the data base is called the mean rebuild time, 

1/jUc 

It is assumed that all faulty disk drives require the same amount of 
repair time. In other words, if the mean response time and the mean 
on-site repair time for a faulty disk drive are c and r, respectively, the 
average total repair time of k faulty disk drives is given as 

- = c + kr. (6) 

4. 1 System states 

For the sake of generality, we assume that n active and m spare disk 
drives are in the simplex system. The various possible states of the 
simplex system are tabulated in Table II. As a result of a disk drive 
failure (State 2) or a headcrash (State 2'), there are (n — 1) active, m 
spare and one faulty disk drives in these two states. At State 3, a spare 
disk drive has been activated and there are consequently n active, 
(m — 1) spare and one faulty disk drives in the simplex system. Similar 
evolution continues until there is no spare disk drive left in State 
(2m + 1). At this point, the simplex system would become inoperative 
for a relatively long period of time if a failure or a headcrash occurred 
resulting in the simplex system being in States (2m 4- 2) or (2m + 2)', 
respectively. 

It can be seen that there are (3m + 3) possible states in the simplex 
systems and all the even-numbered states signify that the normal 
operation of the simplex system is disrupted. 
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Table II — System states of the disk drive 
system 
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4.2 State transition 

The state transition diagram of the simplex system is depicted in 
Fig. 3. A transition takes place between States 1 and 2 or 2' after a disk 
drive fails or a headcrash occurs. By switching over or rebuilding the 
data base, the simplex system is brought back to normal operation at 
State 3. State 3 can return to State 1 after performing the necessary 
repair function. A similar cycle may also be initiated at State 3 and 
terminated at State 5. This repetitive pattern comes to an end at State 
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(2m+2)" 
Fig. 3 — State transition diagram of the disk drive system. 
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(2m + 1), where there is no spare disk drive left. At this point, another 
disk drive failure or headcrash will render the simplex system inoper- 
ative until the faulty drives are repaired. 

It should be pointed out that not all the transitions identified in Fig. 
3 are unique. For example, State 5 can return to State 1 by a direct 
transition or by first returning to State 3. However, the difference in 
the end result between using these two different paths can be shown 
to be small. 

4.3 Mathematical representation 

As before, we assume that both the disk drive failures and the 
headcrashes are Poisson distributed. The time to switchover, to rebuild 
a data base, or to repair a faulty disk drive are exponentially distrib- 
uted. Defining q t as the probability of being in State i of the simplex 
system, we can derive from Fig. 3 a set of first-order differential 
equations: 



Qi 



= —nXqi + ( X M*<72A+1 J + /Wl<72m+2 + (/X m + /i c )<?(2m+2)' 



1 < i < m 
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= hXfq2m+l — jUm+l<72m+2 
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= h\cQ2m+i — (flm + flc)q(2m+2y 


where 





(7) 



and 



A = X f + Ac 



X (<7<2i-n + 92, + q2i) = 1. 



Since each No. 4 ess occupies two disk drives and there are a total 
of n of these disk drives, the probability is 2/n that a No. 4 ess is 
affected when the simplex system is inoperative.* Thus, the unavaila- 
bility of the simplex system from the viewpoint of a single No. 4 ess is 

o m+l 

Us = - I (q 2i + qx)- (8) 

n ,_i 

The availability of the simplex system is therefore given by 

A s = l- U 8 . (9) 



* This is the worst case, as a small No. 4 ess occupies only one disk drive. 
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V. RELIABILITY OBJECTIVE 

Before quantifying the degree of reliability of the cms- IB, we first 
establish the reliability objective required for the trunk maintenance 
operation of the No. 4 ess. During a cms- IB outage, necessary func- 
tions at the trunk test positions would be inoperative, and trunk 
trouble reports would be misplaced. The trunk maintenance operation 
would become so awkward that it would essentially be halted. A 
stoppage in trunk maintenance operation will eventually affect the 
service of the No. 4 ESS. Thus, a cms- IB outage affects the integrity of 
not only the cms- IB itself but also the No. 4 ess. 

Based on these adverse effects, a downtime objective for the cms- IB 
was established. This downtime objective represents the limit above 
which cms- IB outages will have measurable effects on the service of 
the No. 4 ess. To niinimize the adverse impact of cms- IB outages, a 
downtime objective of three minutes per day was judged to be a 
reasonable system goal. 3 It should be pointed out that the stringent 
downtime requirement of three minutes per day is applicable only to 
unexpected outages. Scheduled outages such as preventive mainte- 
nance activities are not included, as they can take place during non- 
critical hours. 

VI. INPUT DATA 

Essentially, two categories of input data are required for the numer- 
ical computation of the reliability model described in Sections III and 
IV. The first category of input data is the mean times between 
hardware failures and between software failures, while the other one 
is the mean times to perform corrective maintenance functions. Wher- 
ever possible, the necessary data were gathered from the existing cms- 
IB sites through the data base maintained by the minicomputer 
reliability group at Columbus, Ohio. For those input data which were 
not available from the data base, we estimated their normative values 
based on the experience of the personnel from the cms development 
group at Holmdel, New Jersey and the cms field support group at 
Merrimack Valley, Massachusetts. However, we will vary the value of 
each of the input data so as to evaluate its impact on the reliability of 
the cms- IB. 

6. 1 Mean time between failures 

Based on the limited downtime data reported by personnel from the 
existing cms- IB sites, some failure statistics on various hardware 
components of the pdp 11/70 computer for the last 12 months has been 
compiled. With the use of these statistics and eq. (3), the mean times 
between simplex hardware failures in the duplex and in the simplex 
systems are calculated to be 40 days and 3.4 months, respectively. 
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No data were available for estimating the mean times between 
headcrashes and between software errors. However, a headcrash rate 
of once every 1.5 years and a minor software error rate of once every 
week were judged to be reasonable, normative values. These values 
will be perturbed in a later section to determine their effects on the 
overall reliability of the cms- IB. The failure statistics to be used in 
this reliability study are summarized in Table III. 

6.2 Corrective maintenance 

The average amount of time devoted to each corrective maintenance 
activity depends to a large extent on the on-site coverage of the cms- 
IB. For example, the hardware repair function is generally covered by 
a maintenance contract with DEC which might govern the maximum 
response time of its repair crew. On the average, a one-half of one day 
response time and a three-hour on-site repair time were experienced. 
The remaining corrective maintenance activities are the responsibility 
of the craftsperson on duty. It was estimated that the average time 
taken to reboot the system, to change over a disk drive, and to rebuild 
a disk pack are 10 minutes, 15 minutes, and 1 hour, respectively. In 
addition to performing the corrective function, these time intervals 
include the detection and the identification of the system unit and 
problem involved. 

VII. RESULTS 

While the cms- IB might serve fewer than six No. 4 esss, our 
reliability results are directed primarily to the worst case with six No. 
4 esss. In the latter case, there are a total of 12 active and two spare 
disk drives in the cms-IB, with each No. 4 ess occupying two active 
disk drives. With the use of the input data defined in Section VI and 
the reliability model characterized in Sections III and IV, the mean 
downtime of the cms-IB is found to be 2.8 minutes a day, and the 
mean time between the cms- IB outages is 4.6 days. It should be 
pointed out that there is a nonzero probability that both the front-end 
and the backup systems are out of service or one of the active disk 
drives experiences a hardware problem after the two spare ones have 
been used. Under this situation, the cms- IB would suffer an outage for 



Table III — Mean time between failures 
statistics 



Item mtbf 



Hardware of duplex system 40 days 

Software 7 days 

Disk drive 3.4 months 

Headcrash 1.5 years 
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as long as 15 hours. However, this extended outage has a probability 
of occurring only once every 2.5 years. 

7. 1 Sensitivity Analysis 

It can be seen that, under the study assumptions, the cms- IB 
satisfies its reliability objective. However, because the reliability is 
close to the limit, the sensitivity of this reliability to changes in some 
of the input data should be determined. 

7.1.1 Mean Response Time 

The sensitivity of the reliability of the cms- IB to the mean response 
time of a repair crew is depicted in Fig. 4. The upper and lower curves 
signify a mean switchover time of 30 and 15 minutes, respectively, 
while the straight line represents the cms- IB downtime objective of 
three minutes per day. As expected, the mean cms- IB downtime is an 
increasing function of the mean response time. The rate of increase 
becomes considerably steeper when the mean response time is in 
excess of one-half day. On the other hand, the mean cms- IB downtime 
is relatively insensitive to the change in the mean switchover time. 
For example, a 10-percent increase in the mean response time consti- 
tutes a 6-percent increase in the mean cms- IB downtime, whereas the 
same amount of increase in the mean switchover time brings about 
only a 1-percent change in the mean cms- IB downtime. 
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Fig. 4 — Sensitivity to mean response time. 
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7. 1.2 Mean time between failures of hardware 

As shown in Fig. 5, the mean cms- IB downtime is inversely propor- 
tional to the mean time between hardware failures in the duplex 
system with a mean response time of one-half day. The mean CMS- IB 
downtime is approaching its asymptotic value rapidly after the mtbf 
becomes larger than 40 days (signified by a vertical bar). For a mean 
response time of one day, the speed of reaching its asymptotic value is 
somewhat slower. To meet the cms- IB downtime objective, a 10- 
percent decrease in the mtbf with an average response time of one- 
half day is tolerable. 

7.1.3 Mean time between failures of software 

The behavior of the mean cms- IB downtime with varying mean 
times between software errors is similar to its hardware counterpart. 
The speed of reaching the asymptotic region, or the insensitive region, 
is dependent on the value of the mean reboot time. Figure 6 indicates 
the required reduction in reboot time to compensate for a higher 
software failure rate and vice versa. This implies that, even if the cms- 
IB software were experiencing a relatively higher failure rate, say, 
once a day, as long as this software error could be rectified within a 
short period of time, say, 30 seconds the cms- IB downtime objective 

could still be met. Figure 6 also indicates that an occurrence of a severe 
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Fig. 6 — Sensitivity to mtbf of software. 

software problem whose reboot time is in the order of hours would not 
meet the cms- IB reliability objective. 

7. 1.4 Mean time between failures of disk drives 

Except for a relatively wider asymptotic region in the vicinity of its 
operating value, the mean time between disk drive failures poses a 
similar impact on the mean cms- IB downtime as the hardware in the 
duplex system. For example, even a 25-percent decrease in the mean 
time between the disk drive failures would not result in any significant 
change in the mean cms- IB downtime. 

7.1.5 Mean time between headcrashes 

The; relationship between the mean time between headcrashes and 
the mean rebuild time is qualitatively similar to that between the 
mean time between software errors and the mean reboot time. How- 
ever, it follows from Fig. 7 that a mean rebuild time in excess of one 
hour would demand a rather high compensation from the headcrash 
rate. 

7.1.6 Number of spare disk drives 

The sensitivity of the mean cms- IB downtime to the number of 
spare disk drives for different mean response times is shown in Fig. 8. 
It is clear that, while more than one spare disk drive is desirable, more 
than three spares are not warranted. 
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VIII. FIELD OF USE GUIDELINES 

The reliability of the cms- IB is clearly dependent on its operating 
characteristics governed by such variables as the mean times between 
various failures and the mean times to perform appropriate corrective 
maintenance functions. The sensitivity analyses conducted in Section 
7.1 indicate that many sets of operating characteristics exist that also 
satisfy the reliability objective required by the No. 4 ess. There are a 
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number of degrees of freedom within which the operating character- 
istics of the cms- IB can be constructed so as to satisfy its reliability 
objective. For example, a more expensive maintenance contract would 
ensure a faster response time of the repair crew, a better designed disk 
drive system would reduce the number of breakdowns and head- 
crashes, a faster magnetic tape drive would shorten the time to rebuild 
data base, and so on. 

It is therefore beneficial to provide some field-of-use guidelines with 
which the most appropriate operating characteristics can be selected 
for each cms-IB. To do this, we will rank-order the degree of impact 
of each of the variables upon the reliability of the cms- IB. The 
quantitative effect of each variable is measured by the percent change 
in the normative value of the variable. It should be noted that, in view 
of the highly nonlinear nature of the operating characteristics of the 
cms- IB, this quantitative measure is valid only in the vicinity of the 
operating region. For example, the effect of the mean response time on 
the reliability of the cms- IB is considerably larger in the 12-hour mean 
response time region than that in the 6-hour one. The degree of impact 
of each of the variables upon the reliability of the cms- IB is summa- 
rized in Table IV. 

It can be seen that the most sensitive, or the most effective, variable 
for improving the reliability of the cms- IB is the mean response time. 
Lower hardware and software failure rates and faster reboot times are 
almost as effective a way of upgrading the system reliability. The mean 
time between software errors and the mean time to reboot have equal 
impact on the system reliability, and their effects are interchangeable. 
Moreover, whenever a major software problem such as a data base 
mutilation occurs, the cms- IB reliability objective will not be satisfied. 

Insofar as system reliability is concerned, Table VI can be used as 
a guideline for selecting the appropriate maintenance contract, speci- 
fying more reliable hardware, etc. 

Table IV — Development 
guidelines 
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IX. CONCLUSION 

Based on the study assumptions reported here, cms- IB is able to 
meet the objective reliability as required by the No. 4 ess. To achieve 
this reliability goal, the following operating characteristics are required 
of the cms- IB: 

(i) Two spare disk drives standing by for a maximum of 12 active 
disk drives. 

(ii) Mean time between duplex hardware failure of 40 days. 

(Hi) Mean time between disk drive failure of 3.5 months with less 
than 15 minutes of mean switchover time to a spare. 

(iv) Mean time between headcrashes of 1.5 years with less than 1 
hour of mean data rebuild time. 

(v) Little or no severe software problems and the mean time between 
minor software errors of one week with less than 10 minutes of mean 
reboot times. 

(vi) Mean response time of less than one-half day and mean on-site 
repair time of less than three hours. 

Each of these six operating characteristics can be achieved by the 
current cms- IB. Furthermore, the reliability of the cms- IB is insensi- 
tive to as much as a 10-percent increase in any one of these operating 
characteristics. 
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